Methods and Devices for Sub-Sampling and Interleaving Multiple Images, EG Stereoscopic

ABSTRACT

Sampled data is packaged in checkerboard format for encoding and decoding. The sampled data may be quincunx sampled multi-image video data (e.g., 3D video or a multi-program stream), and the data may also be divided into sub-images of each image which are then multiplexed, or interleaved, in frames of a video stream to be encoded and then decoded using a standardized video encoder. A system for viewing may utilize a standard video decoder and a formatting device that de-interleaves the decoded sub-images of each frame reformats the images for a display device. A 3D video may be encoded using a most advantageous interleaving format such that a preferred quality and compression ratio is reached. In one embodiment, the invention includes a display device that accepts data in multiple formats.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/148,051 filed Jan. 29, 2009, hereby incorporated by reference inits entirety.

FIELD OF INVENTION

The present invention relates to coding and decoding of digital data,and particularly to digital video data.

DESCRIPTION OF RELATED ART

In recent years, content providers have become considerably interestedin the delivery of stereoscopic (3D) content into the home. Thisinterest is driven by the increased popularity and production of 3Dmaterial, but also the emergence of several stereoscopic devices thatare already available to the consumer. Several systems have beenproposed on the delivery of stereoscopic material to the home.

SUMMARY OF THE INVENTION

The present inventors have realized the need for better encoding anddecoding for 3D and other multi-image video systems. In variousembodiments, the present invention provides methods, systems, andarchitectures for the encoding and decoding of video data. For example,encoding and decoding using checkerboard (CB) interleaved video datawhere data is either “red” or “black” in a format where “red” datafollows “black” data preceding “red” data, etc (the “red” data connotingimage data of a first image, view, or scene, (collectively referred toas “views” plural or “a view” singular) and “black” data connoting imagedata of a second independent view (e.g., of a separate video stream) ora related view (e.g., a second view of a 3D image to be rendered fromboth “red” and “black” data, or another angle of the same view carriedin the “red” data). Extending these concepts, an interleaved video mayinclude any one or more of, for example, multiple images from multipleprogram streams, multiple angles of a same scene, or multiple sets of 3Dviews of a same or different scene, video game, or program, for example.In addition, preparation of frames for encoding may include samplingperformed via quincunx or other sampling techniques. The sampled data isthen arranged in an advantageous format (e.g., a format that considersthe data or capabilities/tendencies of the encoder) which can be astraight checkerboard or groupings of data (e.g., sub-images)interleaved in one or more patterns. Encoding may then be performed, forexample, via MPEG-4 AVC or another encoding technique.

In one embodiment, the present invention provides a method, comprisingthe steps of sub-sampling n images, separating each of the sampledimages into sub-images, packaging the sub-images together into an imageframe, and encoding the single image frame via a video encoder. The stepof sub-sampling comprises, for example, quincunx sampling. The imagesmay comprise, for example, at least one of left and right views of a 3Dimage, multiple views of a same scene, and multiple images (and theimages may also comprise one of depth and occlusion information), or theimages may comprise a group of more than one pixel from one of theimages. The sub-images may comprise data within an image havingcharacteristics similar to a normal image, or the sub-images maycomprise data selected via a pattern from a corresponding image.

The step of separating may comprise, for example, separating the sampledimages based on at least one of rows and columns. The step of separatingmay comprise preparing multiple blocks of nearby data from each of thesampled images. The step of packaging may comprise interleaving thesub-images in a predetermined format. The predetermined format may bechanged adaptively based on, for example, any of the images, the sampledimages, and the separated image samples. The method may further comprisea step of encoding a map identifying the pre-determined format. The mapmay be encoded, for example, in an area of the image frame and thepackaging of sub-images is performed in other areas of the image frame.The map may be encoded as side information and/or made part of an imageframe. The interleaving may comprise, for example, one of a horizontalinterleaving, a vertical interleaving, and a rectangular blockinterleaving.

The step of sub-sampling may comprise quincunx sampling, the step ofseparating may comprise preparing sub-images using one of every otherrow and every other column of a sub-sampled image, and the step ofpackaging may comprise arranging one of row and column based sub-images.The method may further comprise the step of encoding an identifier of anarrangement of sub-images within the image frame. In one embodiment, theidentifier is a code that may be placed in side information of theencoded patterned block. The may also or alternatively be placed in theimage frame.

The step of packaging may comprise packaging according to any of thepatterns described herein or others. The packaging may maintain, forexample, at least one dimension equivalent to a dimension of one of thesub-sampled images. The packaging may comprise, for example, a packagingformat selected for efficient use of resources to be utilized to decodethe encoded image. The packaging may comprise a packaging formatselected for enabling advanced scalability features includingSNR/resolution scalability, and 2D to 3D scalability. The packagingformat may be selected based on available resources, such as processingcapabilities. The packaging may comprise, for example, maintainingpixels of high value for decoding/up-sampling each [sub-image] in closeproximity to each other. The step of encoding may comprise, for example,any one image or video encoding system such as JPEG, JPEG-2000, likeMPEG-2, MPEG-4 AVC, and VC1 encoding.

In other embodiments, the invention may be embodied as a video device,comprising a decoder configured to decode an encoded video signalcomprising more than one image per frame in the video signal, and aformat converter comprising a format converter configured tode-interleave groups of video data interleaved in a frame format in thedecoded video signal wherein the groups of video data comprise one ormore groups of video data from a first image and one or more groups ofvideo data from a second image. The format converter may comprise, forexample, a de-interleaver configured to de-interleave the groups of datafrom multiple interleaving formats. The format converter may comprise,for example, a format reader configured to determine an interleavingformat of the groups of data. The format converter may comprise aselection device configured to select one of an algorithm andspecialized electronics to perform the de-interleaving based on a formatof the interleaved data groups. The format converter may be configured,for example, to de-interleave at least one of horizontal, vertical,block-based, and map-based interleaved groups of video data.

The invention may further comprise an up-converter configured to upconvert the de-interleaved groups of data from at least one of theimages. The up-converted data may be output, for example, as a 2D image.The 2D image may be formatted, for example, in an HDMI compatiblesignal. The up-converted data may comprise data of the first image whichcomprises a first view in a 3D image and data of the second image whichcomprises a second view of the 3D image.

The video device may be part of, for example, at least one of a Blue-RayDVD player, a media player, a set-top box, a cable box, a computer videocard, a tuner, or other electronic device. The decoder may comprise oneof an MPEG-2, MPEG-4 AVC, VC1, and other decoders.

In another embodiment, the invention may be embodied as an encodingsystem, comprising, a sub-sampler configured to sub-sample images of atleast two different views, a formatter configured to select at least onegroup of image data from each view and interleave the groups into asingle image frame of a video stream, and an encoder configured toencode the video stream. The encoder may comprise, for example, anMPEG-4 AVC encoder. The groups of image data comprise, for example,groups of more than one pixel. The formatter may comprise, for example,an even-odd row-column selector and the interleaving groups of imagedata comprise groups comprising at least one of a horizontalre-arrangement, a vertical re-arrangement, an interleaved horizontalre-arrangement, an interleaved vertical re-arrangement, a blockre-arrangement, an interleaved block re-arrangement, verticalinterleaved re-arrangement, and a horizontal interleaved re-arrangement.

The encoder may further comprise a selection device configured to selectan arrangement for interleaving the groups of data. The encoder mayfurther comprise a mapper configured to map an arrangement of data fromthe two images as formatted.

The invention may also be embodied as a media storage having a videostream stored thereon, wherein the video stream comprises interleavedsets of data from at least two views, that, when loaded and read by acorresponding media player, cause the player to decode and thende-interleave the video stream and then format the video stream for adisplay device. The sets of data may comprise, for example, multiplesets of data corresponding to a first view of a 3D image and multiplesets of data corresponding to a second view of the 3D image. The mediastorage may comprise at least one of a memory card, a disk, and physicalproperties of an electromagnetic carrier. Storage contents of the mediastorage may be represented by physical characteristics of at least oneof a memory card, an electromagnetic carrier, and an optical diskcomprise the video stream and are encrypted.

The invention may yet also be embodied as a video encoding system,comprising, a formatter configured to format at least one package ofdata corresponding to a first image, at least one package of datacorresponding to a second image, at least one of a resolution anddynamic range enhancement of the first image, and at least one of aresolution and dynamic range enhancement of the second image into animage data frame of a video stream, and an encoder configured to encodethe formatted first image data and enhancements, second image andenhancements into a video stream for at least one of storage andbroadcast. The encoder may constrain sub-images from performingprediction from samples that correspond to other sub-images. The encodermay constrain sub-images packaged earlier in space from performingprediction from samples that correspond to other sub-images.

The invention may yet further be embodied as a video decoding system,comprising, a decoder configured to decode a data frame of a videostream, wherein the data frame comprises image data from at least twoimages and enhancements for at least one of the images, and are-formatter configured to re-format the decoded image data from atleast one of the images to produce a low resolution version of anoriginal image embodied by the decoded image data. The re-formatter maycomprise, for example, a de-interleaver configured to de-interleave datacorresponding to the at least one image. The re-formatter may be furtherconfigured to discard at least one of the second image data and theenhancements for at least one of the images. The video decoding systemmay further comprise an enhancer configured to utilize at least some ofthe decoded enhancements to enhance the decoded image and produce atleast one of a higher resolution and higher dynamic range image. Theenhancements may be applied to each image progressively and to an extentthe video decoding system is capable of doing so in real-time. Theenhancements may be applied to each image progressively if the videodecoding system is capable of doing so in real-time and an outputdisplay device is capable of displaying the enhanced images.

Portions of both the device and method may be conveniently implementedin programming on a general purpose computer, or networked computers,and the results may be displayed on an output device connected to any ofthe general purpose, networked computers, or transmitted to a remotedevice for output or display. In addition, any components of the presentinvention represented in a computer program, data sequences, and/orcontrol signals may be embodied as an electronic signal broadcast (ortransmitted) at any frequency in any medium including, but not limitedto, wireless broadcasts, and transmissions over copper wire(s), fiberoptic cable(s), and co-ax cable(s), etc.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is a diagram illustrating a checkerboard (CB) multiplexing formataccording to an embodiment of the present invention;

FIG. 2 is a diagram illustrating transform based de-multiplexing of CBdata in a frame mode according to an embodiment of the presentinvention;

FIG. 3 is a diagram illustrating transform based de-multiplexing of CBdata in a field mode according to an embodiment of the presentinvention;

FIG. 4 is a diagram illustrating block motion compensation withoutoverlapping considerations according to an embodiment of the presentinvention;

FIG. 5 is a diagram illustrating transform based de-multiplexing of CBdata in a frame mode according to an embodiment of the presentinvention;

FIG. 6 is a drawing of a video encoder according to an embodiment of thepresent invention;

FIG. 7 is a drawing of a video de-encoder according to an embodiment ofthe present invention;

FIG. 8 is a diagram illustrating utilization of square blocks that maybe extended to diamond or other blocks depending on the nature ofcontent being encoded according to an embodiment of the presentinvention;

FIG. 9 is a diagram of a quincunx sampled image according to anembodiment of the present invention;

FIG. 10 is a diagram illustrating horizontal and vertical re-arrangement(formatting) of quincunx samples to improve coding efficiency accordingto embodiments of the present invention;

FIG. 11 is a diagram illustrating horizontal and vertical “block”re-arrangement (formatting) of quincunx samples to improve codingefficiency according to embodiments of the present invention;

FIG. 12 is a diagram illustrating a variety of arrangements that may beutilized with quincunx sampled data (or extended to other samplingtechniques) according to an embodiment of the present invention;

FIG. 13 is a diagram illustrating a CB arrangement that interleavesblocks of sampled data according to an embodiment of the presentinvention;

FIG. 14 is a diagram illustrating an arrangement and a map that eitherdictates or identifies an arrangement in each sub-region of interleavedsamples according to an embodiment of the present invention;

FIG. 15 is a drawing of a video encoder according to an embodiment ofthe present invention; and

FIG. 16 is a drawing of a video decoder according to an embodiment ofthe present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In one embodiment, the invention extends the MPEG-4 AVC standard to moreappropriately consider the characteristics of the encoded signal,allowing improved coding efficiency and performance. The invention maybe implemented, for example, not only at the encoder but also at thedecoder. Similar extensions may be made to other encoding/decodingstandards, methods, devices, and/or systems. Applications include, forexample, Blu-ray video disks and may also include broadcast and downloadsolutions which are more bandwidth constraints (among others). Theinvention may also be used in a scalable solution that could improve orenhance the current Dolby consumer level 3D video coding system (orother 3D and/or multi-view systems) to full resolution.

The invention in various embodiments is primarily intended for use inDolby (or other) Stereoscopic (3D) format video encoders & decoders, butmay be used in other Dolby and/or non-Dolby specific equipment and/orother types of video (e.g., multi-program, multi-view, multi 3D views,either alone or in combination with others). Applications include, forexample, Blu-ray discs, memory cards, broadcast, satellite, and IPTVsystems, etc.

The present inventors have realized that to ensure rapid adoption of 3Dand other technologies among consumers, a solution should be one thatcan be implemented with minimal or no alteration to existing playbackdevices such as set-top boxes, DVD, and Blu-ray disk players, as well asexisting 3D capable displays. However, converter boxes,hardware/firmware/software modifications, devices and/or displaysspecifically adapted or designed to new or multiple formats are alsoconsistent with the present invention. One possible solution for thedelivery of 3D content without alteration of playback devices is thecreating, coding, and delivering video content information bymultiplexing the two views using a checkerboard arrangement (see FIG.1). Such a system may be implemented using the MPEG-4 AVC/H.264 videocoding standard or other standards (e.g., Microsoft's VC1). However, thestandardized codecs do not consider the nature of the 3D encoded videosignal, resulting in suboptimal coding performance.

In particular, these codecs have been designed and contain tools withprogressive or row interleaved (interlaced) video content in mind (e.g.,only progressive or row interlaced video content). These include toolssuch as motion estimation, motion compensation, transform, andquantization. However, checkerboard interleaved data can have verydifferent characteristics from progressive or interlaced content. Invarious embodiments of the invention, these tools are extended toproperly account for the characteristics of the data and/or thearrangement in which the data is placed, and therefore improve thecoding efficiency, of the content (e.g., content in checkerboardformat).

In one embodiment, video coding efficiency of checkerboard interleavedcontent can be achieved by only modifying the transform and quantizationprocess to be applied on checkerboard de-multiplexed data. Inparticular, as can be seen also from FIG. 2, in this scenario motionestimation and compensation are performed using traditional block basedmethods that do not account for overlapping blocks (FIG. 4). Thisprocess can be justified by the argument that, in general, thecheckerboard multiplexed data are characterized by similar motion.

However, after motion compensation or intra prediction is performed, theresidual data are checkerboard de-multiplexed before transform andquantization. Given the fact that common transform methods employed inexisting codecs are square or orthogonal, de-multiplexing in thisscenario does not happen only in terms of different views but also interms of rows. This would result in 4 blocks that would have to betransformed, e.g., using the 4×4 or 8×8 Integer DCT or other transform,quantized, zig-zag scanned and encoded. In another embodiment, forinterlace (i.e. field) pictures, only vertical de-multiplexing may needto be performed since the data are already in the appropriatearrangement for operating such operations. This process could besignaled at the sequence, picture, slice, macroblock, or block level.The scanning order of the quantized coefficients can be alsoappropriately designed to account for the frequency differences in thehorizontal and vertical axis. In particular, for field content thescanning order of the transformed coefficients is commonly verticallybiased given the difference between horizontal and vertical frequencies.Given, however, the new coding arrangement we introduce, no suchmodification is necessary and the normal (i.e. zig-zag) scanning ordercan still be used.

In an alternative embodiment, the motion estimation and compensationprocesses are also modified apart from the transform, in similar mannerto account for the characteristics of the content. More specificallyboth the reference and source data are rearranged into multiple sets,each set separating the data according to view and parity. This wouldbasically result into four (4) different arrangements (e.g., even/top orodd/bottom left and right views). This can be seen in FIG. 5, whichincludes an illustration of an embodiment of a reference imagede-interleave result, comprising, clockwise, starting at the top leftbox, a set of “x's” from even rows of the left view picture (or CBLT(left top)), a set of “x's” from odd rows of the left view (or CBLB(left bottom)), a set of “o's” from even rows of the right view picture(or CBRT (right top)), and a set of “o's” from odd rows of the rightview picture (or CBRB (right bottom)). A de-interleave in a matchingformat is also illustrated for a source image.

Each arrangement from the source can be matched with any of thearrangements of the reference data for prediction, which can includeboth intra and inter prediction. After the source data are predicted theresidual data are also transformed, quantized, and coded in the samearrangement. This process can be seen as being rather similar to howinterlace encoding is performed where the data are arranged into odd andeven field/line data. However, in the present invention, data arefurther arranged into odd and even column as well. Similar to ourtransform method (which may be used alone or in combination with othertechniques), this method can be signaled for use at the sequence,picture, slice, macroblock, or block level.

The picture level method, for example, can be seen as performing theencoding of 4 different pictures, CBLT, CBLB, CBRT, and CBRB. These fourpictures can reference any previously encoded picture that is availablein the buffer. Default reference list ordering, for obvious reasons isbiased according to the topology of these pictures, i.e., a CBLT picturewould give higher priority to previous CBLT pictures, a CBLB picturewill give higher priority to previous CBLB pictures etc. Each suchpicture can be encoded with existing coding tools, i.e. AVC. When allpictures are decoded, they are then recombined in the frame buffer forfurther processing as a checkerboard image. If disabled, existing, e.g.progressive or interlace, coding methods are utilized such as the onealready available in MPEG-4 AVC or VC1. It should be noted that thevarious methods of signaling enable combinations of legacy methods withour approach at the picture, slice, macroblock, or/and block levelsimilar to what already exists in MPEG-4 AVC for interlace coding.

In a further embodiment, de-blocking of pixel data, using such a methodis applied only across pixels of the same set. An encoder and decoderemploying such methods can be seen in FIG. 6 and FIG. 7 respectively.

In an additional embodiment, given the characteristics of the content,instead of utilizing square or orthogonal blocks for prediction,transform and quantization, we can instead consider diamond blocks (seeFIG. 8, where darkened/red “o” pixel components (diamond shaped) of anexemplary right view picture, and un-darkened “x” pixel componentsdirectly below each darkened/red “o” pixel component (also diamondshaped) of an exemplary left view picture are shown). That is, motionestimation and compensation are now employed with diamond shaped blocksof size N×M, while the transform of the residual data can be performedby using square or orthogonal transforms by first rotating the residualby an appropriate angle (e.g. 45 degrees). De-blocking is performed inthis scenario on the edges of the diamond block data. Furthermore, imageboundaries are processed by appropriately padding the data. The methodcan again be enabled at the sequence, picture, slice, macroblock orblock level and can be combined with any of the previous describedmethods. However, it is preferred, primarily due to complexity andperformance reasons, that this method is considered mainly at thesequence or picture level.

In yet another embodiment, any of the above methods could be utilizedfor the encoding of not only checkerboard interleaved images but also ofresidual data from checkerboard interleaved images, or a combination offour images that are interleaved using a periodic square tiling method.

This invention can be configured as an extension of video coding systemssuch as those based on MPEG-4 AVC.

Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of claims to be included in asubsequently filed utility patent application, the invention may bepracticed otherwise than as specifically described herein.

Referring again to the drawings, wherein like reference numeralsdesignate identical or corresponding parts, and more particularly toFIG. 9 thereof, there is illustrated a sampling method for images anddata referred to as the quincunx sampling.

In quincunx sampling, data is sampled in a quincunx arrangement as isshown in FIG. 9. The benefit of this method, unlike horizontal and/orvertical sampling is that only ˜30% of the information is lost duringthe sampling process, which enables higher quality during thereconstruction of the signal. The method may be used, for example, toproduce samples to be multiplexed into the CB interleaved picture ofFIG. 1. In that case, quincunx sampling is used in part to compress 3Dimages, for example. More specifically, both views of a 3D image mayfirst be quincunx sampled and then interleaved using a checkerboardarrangement prior to compression using a codec (e.g. existing codecssuch as MPEG-2, MPEG-4 AVC, and VC-1 among others).

Although we have shown that it is possible to compress quincunx sampleddata, unfortunately existing compression algorithms are not welldesigned and to some extent are suboptimal to handle theircharacteristics. The resent invention includes different methods toencode such content, which would allow the existing infrastructure to beexploited while achieving improved coding efficiency and performance.This is done by performing various rearrangements of the quincunxsampled data that better fit the content characteristics [and encodingmechanisms].

In particular, we observe that quincunx samples can be separated in oddand even column (or row) data. Odd column data, if seen on their own,have similar characteristics as a normal image even though theirfrequency characteristics may be somewhat different. The same could alsobe said for even column data. We can therefore separate a quincunxsampled image into two sub-images, an odd column sub-image and an evencolumn sub-image. These sub-images contain all information associatedwith the data such as luma and chroma information, transparency anddepth information etc. Obviously, for scalable image systems eachsub-image would also contain all relevant scalability information suchas SNR layers.

Even though we can encode each sub-image separately, it may be desirablefor some environments and/or applications to instead keep the sub-imagestogether as a single image. A possible method, for example is to packagethe two sub-images vertically (FIG. 10 a) or vertically (FIG. 10 b).Nevertheless, for some other applications it is possible to considerinterleaving the two sub-images as well (FIG. 11). The amount ofinterleaving could be fixed or adaptive and could depend on a variety ofrequirements that our system or architecture may have. For example, ifcompression is of higher importance, then the interleaving could be keptto a minimal, therefore better exploiting the signal characteristicsduring compression (e.g. use of the discrete cosine transform/DCT andquantization, prediction etc). If, however, the reconstruction of dataincluding memory access is of higher importance, then some interleavingbetween the views could be used. For example, instead of keeping thedata packed using the original quincunx arrangement, the data can bepacked according to their quincunx parity (odd or even) into rectangularblocks of N×M. In the two most extreme cases, {N=1, M=1} and {N=width/2,M=height/2} where width and height are the width and height of theoriginal non sampled image. These rectangular blocks can be arranged ina variety of ways such as blocks of 4×2 size arranged horizontally (FIG.11 a), or blocks of 2×2 size arranged vertically (FIG. 11 b). In aspecial example, given that most existing video and image codecs useblocks of size 16×16 for prediction, such a block size, or sizes largerthan this size (e.g. 32×32, 32×48, 48×48 etc), could also be used. Notethat in such arrangements it may be desirable to keep one of theresolution dimensions the same as that of the original non-sampledimage, even though that requirement is not necessary.

As we have discussed earlier, a special case of quincunx sampled data isused for 3D applications. In this scenario, two stereo images are firstquincunx sampled and then interleaved together to generate a singlestereo image. Instead of only interleaving these images using a pixellevel checkerboard arrangement (e.g., as in FIG. 1), interleavingmethods may be employed as discussed earlier to better separate the twoimages, therefore better exploiting existing tools for compression. Morespecifically, we can now separate the left and right views into left-odd(Lo), left-even (Le), right-odd (Ro), and right-even (Re) data(Lo|Le|Ro|Re).

In one embodiment, each set of data represents a different sub-image. Inthe case of Lo|Le|Ro|Re, these four sub-images can be tiled together ina variety of arrangements, as shown, for example, in FIG. 12 (otherarrangements and different sizes of the images arranged may also beutilized). The tiled images then represent a new image that can now beencoded using existing or newly developed encoding algorithms. Forexample, we can arrange the four sub-images in the Lo|Le|Ro|Re framearrangement as shown in FIG. 12, or the checkerboard like frame levelarrangement B (Lo|Ro|Re|Le). The sub-images could also be arranged allin a horizontal or vertical sub-image arrangement (arrangements D andE). Other arrangements are also possible. The arrangement type candepend on the application and its requirements. For example, arrangementA provides the benefit that one can immediately reconstruct all samplesfor one view independently from the other, especially if reconstructionto full resolution is required, while the method B may provide benefitsin reorganizing the quincunx data into other arrangements.

In a different embodiment, interleaving could again consider instead ofsingle samples or the entire sub-image, groups of samples, whichessentially comprise a rectangular or even arbitrary block/region.Blocks for example could again be of fixed size M×N (FIG. 13), as wasalso discussed earlier, or an image could be comprised by blocks ofvarying shapes and/or sizes. Such an arrangement could be signaled,through, for example, a metadata method such as a map. The map could befixed for the entire video sequence, or could be adaptive and signaledwhenever necessary. As an example, in FIG. 14, a map with is providedthat provides information of how sub-blocks of size 4×4 are organized interms of interleaving. The same correspondence could apply to allsamples associated with a pixel, e.g. luma and chroma information,transparency, depth/occlusion information etc, but differentarrangements could also be used, including the presence ofmultiple/separate maps, for certain groups or for each different type ofinformation. The separation could also involve different representationsof an image, or different layers (e.g. SNR, bit depth, etc) of an image.

In another embodiment, any of the above methods could be utilized forthe encoding of not only checkerboard interleaved images but also ofresidual data from checkerboard interleaved images, or a combination ofany images that are interleaved using a periodic square tiling method.The method could also be easily extended in the interleaving of multipleimages (beyond 2), including depth/occlusion information.

Finally, the proposed interleaving methods could be used not only whenencoding an image, but also for the generation of prediction images thatcan be utilized in a motion compensated video coding environment.

An encoder that utilizes a format converter that converts a quincunxsampled image or stereo pair into the appropriate format is presented inFIG. 15. The corresponding decoder, that decodes the image and convertsthis format to a different format which may be required for display orother processes is presented in FIG. 16.

Thus the present invention may take many forms. Provided hereinbelow area set of Enumerated Example Embodiments (EEEs), which are exemplaryforms of the invention that are provided as examples. As such the EEEsshould not be viewed as limiting any of the above discussion or anyclaims also presented hereinbelow or in any follow-on continuations,re-uissues, or foreign counterpart patents and/or applications. Theexamples are:

EEE1. A method, comprising the steps of:

sub-sampling n images;separating each of the sampled images into sub-images; andpackaging the sub-images together into an image frame; andencoding the single image frame via a video encoder.

EEE2. The method according to EEE1, wherein the step of sub-samplingcomprises quincunx sampling.

EEE3. The method according to EEE1, wherein the images comprise at leastone of left and right views of a 3D image, multiple views of a samescene, and multiple images.

EEE3A. The method according to EEE3, wherein the images comprise one ofdepth and occlusion information.

EEE4. The method according to EEE1, wherein the sub-images comprise datawithin an image having characteristics similar to a normal image.

EEE5. The method according to EEE1, wherein the step of separatingcomprises separating the sampled images based on at least one of rowsand columns.

EEE6. The method according to EEE1, wherein the sub-images comprise agroup of more than one pixel from one of the images.

EEE7. The method according to EEE1, wherein the sub-images comprise dataselected via a pattern from a corresponding image.

EEE8. The method according to EEE1, wherein the step of separatingcomprises preparing multiple blocks of nearby data from each of thesampled images.

EEE9. The method according to EEE1, wherein the step of packagingcomprises interleaving the sub-images in a predetermined format.

EEE10. The method according to EEE9, wherein the predetermined format ischanged adaptively based on at least one of the images, the sampledimages, and the separated image samples.

EEE11. The method according to EEE1, further comprising the step ofencoding a map identifying the pre-determined format.

EEE11B. The method according to EEE11, wherein the map is encoded in anarea of the image frame and the packaging of sub-images is performed inother areas of the image frame.

EEE12. The method according to Claim EEE11, wherein the map is encodedas side information.

EEE13. The method according to EEE9, wherein the interleaving comprisesone of a horizontal interleaving, a vertical interleaving, and arectangular block interleaving.

EEE14. The method according to EEE1, wherein the step of sub-samplingcomprises quincunx sampling, the step of separating comprises preparingsub-images using one of every other row and every other column of asub-sampled image, and the step of packaging comprises arranging one ofrow and column based sub-images.

EEE15. The method according to EEE1, further comprising the step ofencoding an identifier of an arrangement of sub-images within the imageframe.

EEE16. The method according to EEE1, further comprising the step ofencoding an identifier of an arrangement of sub-images wherein theidentifier is a code placed in side information of the encoded patternedblock.

EEE17. The method according to EEE1, wherein the packaging comprises atleast one of the patterns described above.

EEE18. The method according to EEE1, wherein the packaging maintains atleast one dimension equivalent to a dimension of one of the sub-sampledimages.

EEE19. The method according to EEE1, wherein the packaging comprises apackaging format selected for efficient use of resources to be utilizedto decode the encoded image.

EEE19B. The method according to EEE1, wherein the packaging comprises apackaging format selected for enabling advanced scalability featuresincluding SNR/resolution scalability, and 2D to 3D scalability.

EEE19C. The method according to claim 19B, wherein the packaging formatis selected based on available resources, such as processingcapabilities.

EEE20. The method according to EEE1, wherein the packaging comprisesmaintaining pixels of high value for decoding/up-sampling each[sub-image] in close proximity to each other.

EEE21. The method according to EEE1, wherein the step of encodingcomprises any one image or video encoding system such as JPEG,JPEG-2000, like MPEG-2, MPEG-4 AVC, and VC1 encoding.

EEE22. A video device, comprising:

a decoder configured to decode an encoded video signal comprising morethan one image per frame in the video signal;

a format converter comprising a format converter configured tode-interleave groups of video data interleaved in a frame format in thedecoded video signal wherein the groups of video data comprise one ormore groups of video data from a first image and one or more groups ofvideo data from a second image.

EEE23. The video device according to EEE22, wherein the format convertercomprises a de-interleaver configured to de-interleave the groups ofdata from multiple interleaving formats.

EEE24. The video device according to EEE22, wherein the format convertercomprises a format reader configured to determine an interleaving formatof the groups of data.

EEE25. The video device according to EEE22, wherein the format convertercomprises a selection device configured to select one of an algorithmand specialized electronics to perform the de-interleaving based on aformat of the interleaved data groups.

EEE26. The video device according to EEE22, wherein the format converteris configured to de-interleave at least one of horizontal, vertical,block-based, and map-based interleaved groups of video data.

EEE27. The video device according to EEE22, further comprising anup-converter configured to up convert the de-interleaved groups of datafrom at least one of the images.

EEE28. The video device according to EEE27, wherein the up-converteddata is output as a 2D image.

EEE29. The video device according to EEE28, wherein the 2D image isformatted in an HDMI compatible signal.

EEE30. The video device according to EEE27, wherein the up-converteddata comprises data of the first image which comprises a first view in a3D image and data of the second image which comprises a second view ofthe 3D image.

EEE31. The video device according to EEE22, wherein the video device ispart of at least one of a Blue-Ray DVD player, a media player, a set-topbox, a cable box, a computer video card, a tuner, or other electronicdevice.

EEE32. The video device according to EEE22, wherein the decodercomprises one of an MPEG-2, MPEG-4 AVC, VC1, and other decoders.

EEE33. An encoding system, comprising:

a sub-sampler configured to sub-sample images of at least two differentviews;

a formatter configured to select at least one group of image data fromeach view and interleave the groups into a single image frame of a videostream; and

an encoder configured to encode the video stream.

EEE34. The encoding system according to EEE33, wherein the encodercomprises an MPEG-4 AVC encoder.

EEE35. The encoding system according to EEE33, wherein the groups ofimage data comprise groups of more than one pixel.

EEE36. The encoding system according to EEE33, wherein the formattercomprises an even-odd row-column selector and the interleaving groups ofimage data comprise groups comprising at least one of a horizontalre-arrangement, a vertical re-arrangement, an interleaved horizontalre-arrangement, an interleaved vertical re-arrangement, a blockre-arrangement, an interleaved block re-arrangement, verticalinterleaved re-arrangement, and a horizontal interleaved re-arrangement.

EEE37. The encoder according to EEE33, further comprising a selectiondevice configured to select an arrangement for interleaving the groupsof data.

EEE38. The encoder according to EEE33, further comprising a mapperconfigured to map an arrangement of data from the two images asformatted.

EEE39. A media storage having a video stream stored thereon, wherein thevideo stream comprises interleaved sets of data from at least two views,that, when loaded and read by a corresponding media player, cause theplayer to decode and then de-interleave the video stream and then formatthe video stream for a display device.

EEE40. The media storage according to EEE39, wherein the sets of datacomprise multiple sets of data corresponding to a first view of a 3Dimage and multiple sets of data corresponding to a second view of the 3Dimage.

EEE41. The media storage according to EEE39, wherein the media storagecomprises at least one of a memory card, a disk, and physical propertiesof an electromagnetic carrier.

EEE42. The media storage according to EEE39, wherein storage contents ofthe media storage as represented by physical characteristics of at leastone of a memory card, an electromagnetic carrier, and an optical diskcomprise the video stream and are encrypted.

EEE43. A video encoding system, comprising:

a formatter configured to format at least one package of datacorresponding to a first image, at least one package of datacorresponding to a second image, at least one of a resolution anddynamic range enhancement of the first image, and at least one of aresolution and dynamic range enhancement of the second image into animage data frame of a video stream;an encoder configured to encode the formatted first image data andenhancements, second image and enhancements into a video stream for atleast one of storage and broadcast.

EEE44. The video encoding system according to EEE43, wherein the encoderconstrains sub-images from performing prediction from samples thatcorrespond to other sub-images.

EEE45. The video encoding system according to EEE43, wherein the encoderconstrains sub-images packaged earlier in space from performingprediction from samples that correspond to other sub-images.

EEE46. A video decoding system, comprising:

a decoder configured to decode a data frame of a video stream, whereinthe data frame comprises image data from at least two images andenhancements for at least one of the images;a re-formatter configured to re-format the decoded image data from atleast one of the images to produce a low resolution version of anoriginal image embodied by the decoded image data.

EEE47. The video decoding system according to EEE46, wherein there-formatter comprises a de-interleaver configured to de-interleave datacorresponding to the at least one image.

EEE48. The video decoding system according to EEE46, wherein there-formatter is further configured to discard at least one of the secondimage data and the enhancements for at least one of the images.

EEE49. The video decoding system according to EEE46, further comprisingan enhancer configured to utilize at least some of the decodedenhancements to enhance the decoded image and produce at least one of ahigher resolution and higher dynamic range image.

EEE50. The video decoding system according to EEE49, wherein theenhancements are applied to each image progressively and to an extentthe video decoding system is capable of doing so in real-time.

EEE60. The video decoding system according to EEE49, wherein theenhancements are applied to each image progressively if the videodecoding system is capable of doing so in real-time and an outputdisplay device is capable of displaying the enhanced images.

In describing preferred embodiments of the present invention illustratedin the drawings, specific terminology is employed for the sake ofclarity. However, the present invention is not intended to be limited tothe specific terminology so selected, and it is to be understood thateach specific element includes all technical equivalents which operatein a similar manner. For example, when describing an interleavingtechnique (e.g., block, vertical, horizontal, or others), any otherequivalent interleaving, or variances of those listed, or entirelydifferent interleaving patterns that otherwise address the same issuesdiscussed herein may be substituted therewith. Furthermore, theinventors recognize that newly developed technologies not now known mayalso be substituted for one or more of the described parts of theinvention and still not depart from the scope of the present invention.All other described items, including, but not limited to encoders,sampling, interleaving, decoders, maps, patterns/arrangements/formats,etc should also be considered in light of any and all availableequivalents.

Portions of the present invention may be conveniently implemented usinga conventional general purpose or a specialized digital computer ormicroprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart based on the present disclosure.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the present invention. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical discs, DVD, HD-DVD, Blue-ray, CD-ROMS, CD or DVD RW+/−,micro-drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices (including flash cards, memorysticks), magnetic or optical cards, SIM cards, MEMS, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,and user applications. Ultimately, such computer readable media furtherincludes software for performing the present invention, as describedabove.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including, but not limited to,sampling, identifying sub-images, arranging sub-images, encoding sideinformation in any form related to the interleaving schemes orsub-images relating to the invention, re-formatting after decoding, andthe display, storage, or communication of results according to theprocesses of the present invention.

The present invention may suitably comprise, consist of, or consistessentially of, any of element (the various parts or features of theinvention) and their equivalents as described herein. Further, thepresent invention illustratively disclosed herein may be practiced inthe absence of any element, whether or not specifically disclosedherein. Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as specifically describedherein.

1. A method, comprising the steps of: sub-sampling of full resolutionmulti-view video data into sampled images; separating each of thesampled images into a plurality of sub-images, wherein each sub-imagecomprises image data of the corresponding sampled image selected via apattern; packaging at least the plurality of sub-images of one sampledimage together into a single image frame of a video stream, wherein thepackaging comprises arranging and interleaving the sub-images accordingto an arrangement map that is encoded as side information and/or madepart of an image frame; and encoding the single image frame via a videoencoder.
 2. The method according to claim 1, wherein the arrangement mapprovides information of how sub-blocks of the sub-images are organizedin terms of interleaving.
 3. A method, comprising the steps of:sub-sampling of full resolution multi-view video data into sampledimages; separating each of the sampled images into a plurality ofsub-images, wherein each sub-image comprises image data of thecorresponding sampled image selected via a pattern; packaging at leastthe plurality of sub-images of one sampled image together into a singleimage frame of a video stream, wherein the packaging comprises tilingthe sub-images together in one of a variety of arrangements; andencoding the single image frame via a video encoder, and encoding anidentifier of the arrangement used for tiling the sub-images.
 4. Themethod according to claim 3, wherein the identifier is a code that isplaced inside information of an encoded patterned block and/or that isplaced in the image frame.
 5. The method according to claim 1, whereineach sampled image is obtained by sub-sampling only a single view fromthe multi-view video data.
 6. The method according to claim 1, whereinthe step of sub-sampling comprises quincunx sampling of full resolutionimages comprising left view image data and right view image data intoquincunx sampled images, wherein each quincunx sampled image comprisesonly one of a sampled left view image and a sampled right view image. 7.The method according to claim 6, wherein the packaging step comprisespackaging the plurality of sub-images of a first sampled image and theplurality of sub-images of a second sampled image together into a singleimage frame, wherein the first and second sampled images correspond todifferent views in the multi-view video data.
 8. The method according toclaim 1, wherein the step of separating comprises separating eachsampled image into sub-images by selecting image data of determined rowsand/or columns of the sampled image.
 9. The method according to claim 1,wherein the pattern separates each sampled image into an even rowsub-image and an odd row sub-image, or wherein the pattern separateseach sampled image into an even column sub-image and an odd columnsub-image.
 10. The method according to claim 1, wherein the step ofseparating comprises preparing multiple blocks of image data from eachof the sampled images.
 11. The method according to claim 1, wherein thearrangement map is fixed for the entire video sequence.
 12. The methodaccording to claim 1, wherein the arrangement map is changed adaptivelybased on at least one of the images, the sampled images, and theseparated image samples.
 13. The method according to claim 1, whereinthe arrangement map is encoded in a portion of the image frame and thepackaging of sub-images is performed in other portions of the imageframe.
 14. The method according to claim 9, wherein the packagingcomprises a horizontal arrangement of the two sub-images of one view, orwherein the packaging comprises a vertical arrangement of the twosub-images of one view.
 15. The method according to claim 9, whereineach of the sub-images of one view comprises one or more rectangularblocks, and wherein the blocks of the two sub-images of one view arearranged in an interleaved combination.
 16. The method according toclaim 1, wherein the packaging maintains at least one dimensionequivalent to a dimension of one of the sub-sampled images.
 17. Themethod according to claim 1, wherein the arrangement map is selected forenabling advanced scalability features including SNR/resolutionscalability, and 2D to 3D scalability.
 18. The method according to claim1, wherein the step of encoding comprises any one image or videoencoding system such as JPEG, JPEG-2000, like MPEG-2, MPEG-4 AVC, andVC1 encoding.
 19. A video device, comprising: a decoder configured todecode an encoded video signal comprising more than one image per framein the video signal; a format converter comprising a format converterconfigured to de-interleave groups of video data interleaved in a frameformat in the decoded video signal wherein the groups of video datacomprise one or more groups of video data from a first image and one ormore groups of video data from a second image, wherein the formatconverter comprises a de-interleaver configured to de-interleave thegroups of data from multiple interleaving formats according to one of(1) an arrangement map that is encoded as side information and/or madepart of an image frame, or (2) an identifier that is encoded in thevideo signal and that identifies one of a plurality of tilingarrangements.
 20. The video device according to claim 19, wherein theformat converter comprises a selection device configured to select oneof an algorithm and specialized electronics to perform thede-interleaving based on a format of the interleaved data groups. 21.The video device according to claim 19, wherein the format converter isconfigured to de-interleave at least one of horizontal, vertical,block-based, and map-based interleaved groups of video data.
 22. Thevideo device according to claim 19, further comprising an up-converterconfigured to up convert the de-interleaved groups of data from at leastone of the images.
 23. The video device according to claim 22, whereinthe up-converted data comprises data of the first image which comprisesa first view in a 3D image and data of the second image which comprisesa second view of the 3D image.
 24. The video device according to claim19, wherein the video device is part of at least one of a Blue-Ray DVDplayer, a media player, a set-top box, a cable box, a computer videocard, a tuner, or other electronic device.
 25. An encoding system,configured to perform the method of claim
 1. 26. An encoding system,comprising: a sub-sampler configured to sub-sample images of at leasttwo different views; a formatter configured to select at least one groupof image data from each view and interleave the groups into a singleimage frame of a video stream, wherein the selecting and theinterleaving is based on one of (1) an arrangement map that is encodedas side information and/or made part of an image frame, or (2) anidentifier that is encoded in the video signal and that identifies oneof a plurality of tiling arrangements; and an encoder configured toencode the video stream.
 27. The encoding system according to claim 26,wherein the encoder comprises an MPEG-4 AVC encoder.
 28. The encodingsystem according to claim 26, wherein the groups of image data comprisegroups of more than one pixel.
 29. The encoding system according toclaim 26, wherein the formatter comprises an even-odd row-columnselector and the interleaving groups of image data comprise groupscomprising at least one of a horizontal re-arrangement, a verticalre-arrangement, an interleaved horizontal re-arrangement, an interleavedvertical re-arrangement, a block re-arrangement, an interleaved blockre-arrangement, vertical interleaved re-arrangement, and a horizontalinterleaved re-arrangement.
 30. The encoder according to claim 26,further comprising a selection device configured to select anarrangement for interleaving the groups of data.